1. Field
The technology described in this patent document relates to the field of video signal processing. More specifically, systems and methods are disclosed for performing motion compensated frame rate conversion of an input video signal at a first frame rate to an output video signal at a second frame rate.
2. Related Art
The picture refresh rates of modern displays (e.g., LCD, PDP, DLP, etc.) range from 50 Hz to 120 Hz, while the picture rates of video sources can be either 50 Hz/60 Hz interlaced, or 24/25/30/48/50/60 Hz progressive, or others. Thus, picture rate conversion is oftentimes necessary to address the disparity between the various source picture rates and the various display rates. For interlaced video sources, such as standard broadcast TV sources, picture rate conversion is normally performed after the de-interlacing function generates a sequence of video frames, and thus the term “frame rate conversion” (FRC) is often used in the literature to describe this conversion process.
A number of FRC methods are known in this field. Among these known methods are three simple linear processing methods: (1) frame repeat, (2) frame drop, and (3) temporal averaging. In the frame repeat method, a video source of 50 frames per second (fps) is converted to 60 fps by simply keeping all the original frames and inserting a repeated frame for every five original frames. In the frame drop method, a video source of 60 fps is converted to 50 fps by dropping every sixth original frame. And in the temporal averaging method a video source of 60 fps is converted to 120 fps by keeping the original frames and generating additional new frames by averaging every two consecutive original frames.
These methods may work well for video sources with static scenes or very slow motion, but for video sources with moderate to fast motion, these methods produce noticeable visual artifacts, such as motion judder and motion blur, especially on large displays. To avoid such artifacts, motion compensated frame rate conversion (MC-FRC) has been proposed and adopted in some commercial products. In MC-FRC, new frames are interpolated from their temporally preceding and following original frames, where the interpolation is along the motion trajectories of the objects in the original frames. MC-FRC has the potential of producing significantly better visual quality than the three aforementioned simple FRC methods.
Although MC-FRC has the potential of producing significantly better visual quality than other FRC methods, a number of challenging issues must be carefully addressed, to realize this potential. First, MC-FRC requires true motion estimation between the original frames in an input video sequence. An incorrect motion description for an object in the sequence may result in the object being put at an incorrect place in the interpolated frame and this may cause noticeable, and undesirable visual artifacts. Second, it is difficult, and even impossible in some cases, to find the true motion for some objects in a video sequence, due to various reasons such as inadequate temporal sampling rate of the video source, noise in the video source, and occlusions where an object may be covered or uncovered from one input frame to the next. Therefore, it is necessary to have a robust fall-back scheme for generating the new frames that does not exhibit noticeable visual artifacts. Third, MC-FRC tends to have high computational and storage complexity. For a cost-efficient solution, the complexity of the MC-FRC method should be constrained.